{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rHLcriKWLRe4"
   },
   "source": [
    " # Pandas 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QvJBqX8_Bctk"
   },
   "source": [
    "**学习目标：**\n",
    "  * 大致了解 *pandas* 库的 `DataFrame` 和 `Series` 数据结构\n",
    "  * 存取和处理 `DataFrame` 和 `Series` 中的数据\n",
    "  * 将 CSV 数据导入 pandas 库的 `DataFrame`\n",
    "  * 对 `DataFrame` 重建索引来随机打乱数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TIFJ83ZTBctl"
   },
   "source": [
    " [*pandas*](http://pandas.pydata.org/) 是一种列存数据分析 API。它是用于处理和分析输入数据的强大工具，很多机器学习框架都支持将 *pandas* 数据结构作为输入。\n",
    "虽然全方位介绍 *pandas* API 会占据很长篇幅，但它的核心概念非常简单，我们会在下文中进行说明。有关更完整的参考，请访问 [*pandas* 文档网站](http://pandas.pydata.org/pandas-docs/stable/index.html)，其中包含丰富的文档和教程资源。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s_JOISVgmn9v"
   },
   "source": [
    " ## 基本概念\n",
    "\n",
    "以下行导入了 *pandas* API 并输出了相应的 API 版本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "aSRYu62xUi3g",
    "outputId": "7937e84f-83bb-44d2-f305-58b215558433"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "daQreKXIUslr"
   },
   "source": [
    " *pandas* 中的主要数据结构被实现为以下两类：\n",
    "\n",
    "  * **`DataFrame`**，您可以将它想象成一个关系型数据表格，其中包含多个行和已命名的列。\n",
    "  * **`Series`**，它是单一列。`DataFrame` 中包含一个或多个 `Series`，每个 `Series` 均有一个名称。\n",
    "\n",
    "数据框架是用于数据操控的一种常用抽象实现形式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fjnAk1xcU0yc"
   },
   "source": [
    " 创建 `Series` 的一种方法是构建 `Series` 对象。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DFZ42Uq7UFDj",
    "outputId": "cbef69c6-7f29-4b73-a727-034f209053b9"
   },
   "outputs": [],
   "source": [
    "pd.Series(['San Francisco', 'San Jose', 'Sacramento'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U5ouUp1cU6pC"
   },
   "source": [
    " 您可以将映射 `string` 列名称的 `dict` 传递到它们各自的 `Series`，从而创建`DataFrame`对象。如果 `Series` 在长度上不一致，系统会用特殊的 [NA/NaN](http://pandas.pydata.org/pandas-docs/stable/missing_data.html) 值填充缺失的值。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "avgr6GfiUh8t",
    "outputId": "3b02a491-be98-405c-e85a-9b47319d405d"
   },
   "outputs": [],
   "source": [
    "city_names = pd.Series(['San Francisco', 'San Jose', 'Sacramento'])\n",
    "population = pd.Series([852469, 1015785, 485199])\n",
    "\n",
    "pd.DataFrame({'City name': city_names, 'Population': population})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oa5wfZT7VHJl"
   },
   "source": [
    " 但是在大多数情况下，您需要将整个文件加载到 `DataFrame` 中。下面的示例加载了一个包含加利福尼亚州住房数据的文件。请运行以下单元格以加载数据，并创建特征定义：\n",
    "### 数据加载和查看"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "id": "av6RYOraVG1V",
    "outputId": "e6484a75-59d4-44fb-e853-5439cad1eed9"
   },
   "outputs": [],
   "source": [
    "# 使用DataFrame.head(),显示前5行数据。\n",
    "df = pd.read_csv('./dataset/Pokemon.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "id": "av6RYOraVG1V",
    "outputId": "e6484a75-59d4-44fb-e853-5439cad1eed9"
   },
   "outputs": [],
   "source": [
    "# 也可以使用display函数\n",
    "from IPython.display import display \n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "id": "av6RYOraVG1V",
    "outputId": "e6484a75-59d4-44fb-e853-5439cad1eed9"
   },
   "outputs": [],
   "source": [
    "# 显示DataFrame数据的每列的数据类型和非空数据\n",
    "df.info() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 显示DataFrame的描述性统计数据\n",
    "df.describe() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 绘图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w9-Es5Y6laGd"
   },
   "source": [
    " *pandas* 的另一个强大功能是绘制图表。例如，借助 `DataFrame.hist`，您可以快速了解一个列中值的分布："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 316
    },
    "id": "nqndFVXVlbPN",
    "outputId": "879dda92-7102-4198-efa9-42b938bcc4d9"
   },
   "outputs": [],
   "source": [
    "# 使用hist针对DataFrame的某个特征画直方图\n",
    "df.hist('Total',bins = 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XtYZ7114n3b-"
   },
   "source": [
    " ## 访问DataFrame中的数据\n",
    "\n",
    "您可以使用熟悉的 Python dict/list 指令访问 `DataFrame` 数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_TFm7-looBFF",
    "outputId": "feb46674-17e0-443d-cff4-bb743a1b23e2"
   },
   "outputs": [],
   "source": [
    "# 访问DataFrame中的某一列\n",
    "print(type(df['Total']))\n",
    "df['Total']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "V5L6xacLoxyv",
    "outputId": "e96442bd-2b89-4022-c5a9-7300d8260754"
   },
   "outputs": [],
   "source": [
    "print(type(df['Total'][0]))\n",
    "df['Total'][0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 128
    },
    "id": "gcYX1tBPugZl",
    "outputId": "735a9470-7b54-4f16-d309-b7bc2457315c"
   },
   "outputs": [],
   "source": [
    "# 数据切片，选择指定的行数\n",
    "print(type(df[0:5]))  # 前5行\n",
    "df[0:5]    # 等价于df.iloc[0:5]，df.iloc[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 满足一定条件的数据选择 用[]来写条件\n",
    "# 选择Type1是Grass的数据展示\n",
    "df[df['Type 1'] == 'Grass']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style='color:red'>练习</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# 填写你的代码\n",
    "# 选择 total > 700 且 Attach > 100 的数据\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "# 参考答案\n",
    "df[(df['Total'] >700) & (df['Attack'] > 100)]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "65g1ZdGVjXsQ"
   },
   "source": [
    " 此外，*pandas* 针对高级[索引和选择](http://pandas.pydata.org/pandas-docs/stable/indexing.html)提供了极其丰富的 API（数量过多，此处无法逐一列出）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RM1iaD-ka3Y1"
   },
   "source": [
    " ## 操控数据\n",
    "\n",
    "您可以向 `Series` 应用 Python 的基本运算指令。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XWmyCFJ5bOv-",
    "outputId": "1217db60-56a6-41f6-828f-91726326a177"
   },
   "outputs": [],
   "source": [
    "df['Total'][:5], df['Total'][:5] / 1000."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TQzIVnbnmWGM"
   },
   "source": [
    " [NumPy](http://www.numpy.org/) 是一种用于进行科学计算的常用工具包。*pandas* `Series` 可用作大多数 NumPy 函数的参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ko6pLK6JmkYP",
    "outputId": "90a72458-ffd8-464e-8bc3-1c84832f19bf"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.log(df['Total'][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xmxFuQmurr6d"
   },
   "source": [
    " 对于更复杂的单列转换，您可以使用 `Series.apply`。像 Python [映射函数](https://docs.python.org/2/library/functions.html#map)一样，`Series.apply` 将以参数形式接受 [lambda 函数](https://docs.python.org/2/tutorial/controlflow.html#lambda-expressions)，而该函数会应用于每个值。\n",
    "\n",
    "下面的示例创建了一个指明 `population` 是否超过 100 万的新 `Series`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Fc1DvPAbstjI",
    "outputId": "818c5dfb-672b-4458-aa88-5b0505c50bf5"
   },
   "outputs": [],
   "source": [
    "df['Total'][:5], df['Total'][:5].apply(lambda x: x > 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 上述简单的命令 等价于：\n",
    "df['Total'][:5] > 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZeYYLoV9b9fB"
   },
   "source": [
    " \n",
    "`DataFrames` 的修改方式也非常简单。例如，以下代码向现有 `DataFrame` 添加了两个 `Series`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "0gCEX99Hb8LR",
    "outputId": "a8b710bb-9726-4c95-d29b-06fd3a889d04"
   },
   "outputs": [],
   "source": [
    "df['new_col_1'] = df['Total']/100.\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6qh63m-ayb-c"
   },
   "source": [
    "<h2 style='color:red'>练习</h2>\n",
    "\n",
    "通过添加一个新的布尔值列`new_col_2`（当且仅当以下*两项*均为 True 时为 True）修改 `df` 表格：\n",
    "\n",
    "  * Name 以 'B'开头。\n",
    "  * Type2 为空。\n",
    "\n",
    "**注意：**布尔值 `Series` 是使用“按位”而非传统布尔值“运算符”组合的。例如，执行*逻辑与*时，应使用 `&`，而不是 `and`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "zCOn8ftSyddH",
    "outputId": "b0516511-3854-4695-fdfa-697c6c40e16e"
   },
   "outputs": [],
   "source": [
    "# 此处填写你的代码\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "# 参考答案\n",
    "df['new_col_2'] = (df['Name'].str.startswith('C')) & (df['Type 2'].isnull())\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f-xAOJeMiXFB"
   },
   "source": [
    " ## 索引\n",
    "`Series` 和 `DataFrame` 对象也定义了 `index` 属性，该属性会向每个 `Series` 项或 `DataFrame` 行赋一个标识符值。\n",
    "\n",
    "默认情况下，在构造时，*pandas* 会赋可反映源数据顺序的索引值。索引值在创建后是稳定的；也就是说，它们不会因为数据重新排序而发生改变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2684gsWNinq9",
    "outputId": "dc091312-3b6d-4fbd-e28b-539cfe624a6b"
   },
   "outputs": [],
   "source": [
    "print(df.index)\n",
    "df.index.values[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hp2oWY9Slo_h"
   },
   "source": [
    " 调用 `DataFrame.reindex` 以手动重新排列各行的顺序。例如，以下方式与按城市名称排序具有相同的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "sN0zUzSAj-U1",
    "outputId": "7b1b6a02-2666-407f-cf86-154609399fa6"
   },
   "outputs": [],
   "source": [
    "df.reindex([2, 0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-GQFz8NZuS06"
   },
   "source": [
    " 重建索引是一种随机排列 `DataFrame` 的绝佳方式。在下面的示例中，我们会取用类似数组的索引，然后将其传递至 NumPy 的 `random.permutation` 函数，该函数会随机排列其值的位置。如果使用此重新随机排列的数组调用 `reindex`，会导致 `DataFrame` 行以同样的方式随机排列。\n",
    "尝试多次运行以下单元格！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "mF8GC0k8uYhz",
    "outputId": "4eab2c1b-d057-444c-a9fb-f0573ff360fe"
   },
   "outputs": [],
   "source": [
    "df.reindex(np.random.permutation(df.index))[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fSso35fQmGKb"
   },
   "source": [
    " 有关详情，请参阅[索引文档](http://pandas.pydata.org/pandas-docs/stable/indexing.html#index-objects)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8UngIdVhz8C0"
   },
   "source": [
    "`reindex` 方法允许使用未包含在原始 `DataFrame` 索引值中的索引值。请试一下，看看如果使用此类值会发生什么！您认为允许此类值的原因是什么？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "PN55GrDX0jzO",
    "outputId": "a301b49b-7c7d-46fb-c227-5976b9b64eb0"
   },
   "outputs": [],
   "source": [
    "df.reindex([8, 9, 800,801])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8oSvi2QWwuDH"
   },
   "source": [
    " 如果您的 `reindex` 输入数组包含原始 `DataFrame` 索引值中没有的值，`reindex` 会为此类“丢失的”索引添加新行，并在所有对应列中填充 `NaN` 值： \n",
    " \n",
    " 这种行为是可取的，因为索引通常是从实际数据中提取的字符串（请参阅 [*pandas* reindex 文档](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reindex.html)，查看索引值是浏览器名称的示例）。\n",
    "\n",
    "在这种情况下，如果允许出现“丢失的”索引，您将可以轻松使用外部列表重建索引，因为您不必担心会将输入清理掉。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2l82PhPbwz7g"
   },
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "JndnmDMp66FL",
    "YHIWvc9Ms-Ll",
    "TJffr5_Jwqvd"
   ],
   "name": "intro_to_pandas.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
